† Corresponding author. E-mail:
Project supported by the National Natural Science Foundation of China (Grant No. 51972316), Open Project of State Key Laboratory of ASIC & System (Grant No. 2019KF006), Zhejiang Provincial Natural Science Foundation of China (Grant No. LR18F040002), and Program for Ningbo Municipal Science and Technology Innovative Research Team, China (Grant No. 2016B10005).
Rapid developments in artificial intelligence trigger demands for perception and learning of external environments through visual perception systems. Neuromorphic devices and integrated system with photosensing and response functions can be constructed to mimic complex biological visual sensing behaviors. Here, recent progresses on optoelectronic neuromorphic memristors and optoelectronic neuromorphic transistors are briefly reviewed. A variety of visual synaptic functions stimulated on optoelectronic neuromorphic devices are discussed, including light-triggered short-term plasticities, long-term plasticities, and neural facilitation. These optoelectronic neuromorphic devices can also mimic human visual perception, information processing, and cognition. The optoelectronic neuromorphic devices that simulate biological visual perception functions will have potential application prospects in areas such as bionic neurological optoelectronic systems and intelligent robots.
With the rapid development of artificial intelligence (AI) and Internet of things (IoTs), special requirements are arisen for computation speed and energy consumption.[1,2] Unfortunately, it is hard to satisfy the increasing demands of computation resources with the coming of information explosion by the traditional computation systems based on von Neumann architectures, where the physical separation of memory module and processing module brings about limited data transmission rate and huge energy consumption.[3,4] As comparison, our brain consists of ∼ 1011 neurons and ∼ 1015 synapses. Neurons and synapses are the basic units of brain memory and information processing.[5] They can execute computation in a parallel model. Thus, unstructured tasks such as perception, learning, thinking, memory, and decision-making can be fulfilled in an energy efficient pattern due to the highly parallel, event-driven, and energy-efficient architectures.[6,7] Therefore, brain inspired computation configurations are proposed to overcome the von Neumann bottleneck. Moreover, neuromorhpic devices are considered as a feasible technical way to realize the brain-like computation in artificial neural networks (ANNs) and artificial intelligence.[8] In the last decade, two-terminal resistance switching devices (e.g., memristors,[9–12] phase change memories[13]) and three-terminal transistors (e.g., ferroelectric transistor,[14] electrolyte gated transistors,[15–22] memtransistors[23]) have been proposed for neuromorhpic device applications. Several biological synaptic plasticity behaviors have been mimicked, including paired pulse facilitates (PPF), synaptic filtering, spiketiming dependent plasticity (STDP), metaplasticity, etc. Moreover, neural calculations have been simulated successfully, including pattern memory,[24,25] pattern recognition,[26,27] reservoir computing,[28] spatiotemporal dynamic logic,[15,16,29–31] etc.
A peripheral nervous system can sense and respond to external stimuli such as light, sound, pressure, and chemicals. This sensation information can be transferred to the central nervous system.[32] Previously, pressure sensors[33] and electronic skin (E-skin) devices[34] have successfully simulated the touch response of human skin. Recently, tactile sensing characteristics have been achieved by connecting pressure sensor with neuromorphic transistor or neuromorphic memristor, possessing the functions of signal conversion and information processing.[10,11] Such bionic tactile sensing systems based on neuromorphic devices will have broad application prospects in low-cost bionic smart field information sensing and intelligent identification. We receive most of outer information through our visual perception systems.[35] The photoreceptors in the retina receive light inputs and convert them into electrical signals. The optic nerves, composed of various neurons and adjacent synapses, can transmit the electrical stimuli to the visual cortex. Recently, designing artificial visual perception system is attracting increasing interests. The main functions of the visual perception system include perception, learning, and memory of external information. Kwon et al.[36] designed a light-adjusted optoelectronic neuromorphic circuit consisted of a photovoltaic divider and an ionotronic synaptic transistor. The photovoltaic divider and synaptic transistor act as artificial retina and optic nerve, respectively. The light-adaptable synaptic functions of the biological visual perception system have been simulated successfully. In addition, photodetectors can also simulate artificial photoreceptors to obtain visual information by converting light signals into electrical signals.[37,38] At the same time, photoelectric neuromorphic devices with photosensitivity can directly convert light stimuli into post-synaptic signals to simulate visual perception learning systems. Recently, optoelectronic neuromorphic devices based on two-terminal memristors or three-terminal transistors have been demonstrated using different materials, such as metal oxides,[39,40] carbon nanotubes,[41] graphene,[42] and MoS2.[43] Compared with electrical signals, optical signals can be regarded as additional terminals, which greatly enrich the degree of freedom in regulating the synaptic plasticity. Therefore, bionic systems that combine optoelectronic neuromorphic devices, bionic prostheses, and soft robotics will greatly advance the development of artificial intelligence systems.
Here, we describe the optoelectronic characteristics of two-terminal memristors and three-terminal transistors. Photo-illumination triggered synaptic responses and neural cognitive behaviors are discussed. Later, integration of artificial visual perception and artificial biological systems is described, reflecting the trend of future bionic neural networks. Finally, a short outlook is provided. Table
The concept of memristor was first proposed by Professor Chua from the University of California, Berkeley in 1971.[66] In 2008, HP laboratories successfully manufactured TiO2 based memristor.[67] Memristor is a two-terminal device with a three-layer structure, including top electrode (TE), bottom electrode (BE), and resistive switching (RS) layer. The resistance state can be changed by the amount of charges flowing through the RS layer. Memristors have advantages including simple physical structure, small size, non-volatile properties, memory capacity, low energy consumption, and good scalability. Up to date, various types of memristors have been reported, such as phase change memories (PCMs),[68] resistive switch memories,[69,70] etc. Due to the non-linear electrical properties, memristors have been proposed for artificial synapse and neuromorphic engineering applications.
As an interesting optoelectronic technology, photonic memories have attracted increasing attention in the last few years. Such devices have great potentials to overcome the von Neumann bottleneck. Moreover, integrated systems consisted of arrayed photonic memories have attracted increasing interests for adoption as building blocks in the biospired vision system.[71]
ZnO is a typical wide band gap semiconductor material with a band gap of ∼ 3.37 eV. It has high exciton binding energy and excellent selective absorption of ultraviolet (UV) light. Thus, it has broad application prospects in ultraviolet detectors due to the significant photoelectric performances. As typical functional materials in microelectronic field, ZnO has also been proposed as RS layer in memristor. Recently, ZnO nanorods have been proposed for optically modulated artificial synapse applications.[44] The high resistance state (HRS) of the device is non-volatile, while the low resistance state (LRS) is volatile. This volatile characteristic is beneficial for simulating the short-term synaptic plasticity behavior. The HRS and LRS overlap immediately when exposing to UV illumination, as shown in the yellow part in Fig.
Furthermore, ZnO1 – x/AlOy heterojunction based photoelectric memristor has also been proposed with ITO/ZnO1 – x/AlOy/Al stacks.[39] Here, the AlOy layer is naturally formed on Al. The device shows typical current–voltage (I–V) sweeping curves in the dark and under UV-light illumination, displaying LRS and HRS. The operation mechanism is related to the electron trapping/detrapping at the trapping sites in the AlOy layer. Under negative bias, an electron depletion layer is formed in the AlOy layer, realizing a reset process from the LRS to the HRS. Under positive bias, the depletion layer will get narrow, which causes LRS. While the UV-light illumination modulates the resistive behaviors due to photogenerated holes and persistent photoconductivity effects. Thus, the device can demonstrate photoelectric plasticity by setting light pulse stimulation at different pulse width, pulse intensity, pulse number, and pulse frequency. Figure
Two-dimensional (2D) transition metal dichalcogenides (TMDs) are considered as promising candidates for the next generation optoelectronics and photodetection due to their unique electrical, mechanical, and optical properties.[45,73] Especially, they have also been proposed for photonic memristor applications with light illumination at near-infrared region (NIR) due to the small band gap, showing great potentials in memristive synapses. Due to the combination of photonic and electric neuromorphic functions, they would have prospectives in synthetic retinas and optoelectronic interfaces. Zhai et al.[43] used a heterostructure formed with upconversion nanoparticles (UCNPs) and molybdenum disulfide (MoS2) to achieve optically tunable memory performance under NIR illumination. The upconversion material has excellent light response performances in the NIR due to its photon conversion characteristics.[74,75] With the increased light intensity, both the SET voltage and the RESET voltage gradually decrease. While the ON current and OFF current increase correspondingly. Figure
In human visual system, eyes can receive image information from retina. And the optic nerve can transfer visual signals to the visual center to form visual memory. Figure
These optical-assistance storage systems can reapear the stored image information, and provide a novel approach to imitate human echoic memory and haptic memory. Moreover, the flexible visual memory device will have potentials in wearable electronics, electronic eyes, etc.
Transistors have also been proposed for neuromorphic device applications as early as in 1996 by Carver Mead[77] Due to unique characteristics of the three-terminal transistor, it has certain advantages over two-terminal device in terms of neuromorphic applications. Firstly, the information transfer and learning process can be executed synchronously. Second, the synaptic performance can be regulated by adding additional gates to amplify signals and reduce energy consumption. Thirdly, multi-gates can be integrated for realizing dendritic algorithm.[23,31,78,79] Therefore, three-terminal transistor based neuromorphic devices would act as fundamental building blocks in the field of synaptic bionics and neuromorphic computing. Especially, phototransistor would facilitate the co-regulation of semiconductor channel conductances by light and electrical stimuli. Thus, they would also have great potentials in artificial visual neuromorphic system.
The transmission of information between neurons depends on the delivery of neurotransmitters between synaptic clefts. The release of neurotransmitters leads to different kinds of synaptic functions, including synaptic potentiation and depression, excitatory post-synaptic current, paired pulse facilitation, synaptic filtering, short-term plasticity, long-term plasticity, etc. Thus, one of the main tasks for phototransistors in neuromorphic engineering applications would be the realization of these basic synaptic functions. In phototransistors, light stimuli are regarded as pre-synaptic spikes to trigger the synaptic responses. With smart designs and strategies, basic synaptic response and unique neural functions have been mimicked on these kinds of neuromorphic transistors.
Recently, layered 2D materials have been paid attention for neuromorphic system applications.[80,82,83] Kim et al.[80] designed MoS2 phototransistors for optical synaptic device applications, as schematically shown in Fig.
As discussed in the previous part, quantum dots (QDs) have been proved to be good choices in functional optoelectronic device applications. Photoresponse behaviors could be modulated by the sizes and the compositions of the QDs. The strategies have also great potentials in artificial light-sensitive synaptic devices. Recently, Huang et al.[81] reported a light-stimulated synaptic transistors (LSSTs) by blending inorganic halide perovskite quantum dots (IHP QDs) and organic semiconductors (OSCs) together. The device demonstrates improved charge separation efficiency of the photoexcited charges. Moreover, the IHP QDs/OSCs hybrid film demonstrates delayed decay in photocurrent, which is important for achieving synaptic performances. A light pulse (500 nm, 0.1 mW/cm2, pulse width 500 ms) induces a typical EPSC response with peak EPSC value of ∼ 1.3 nA, as shown in Fig.
It is interesting to note here that ionic-liquid or ionic-gel based electrolyte gated transistors (EGTs) can operate at low-voltage due to the unique interfacial ionic gating effects.[17,88–90] Interestingly, ion gating behaviors in EGTs are quite similar to those in biological synapse, which means that unique ion modulation behavior in EGTs could establish bridges between iontronic devices and neuromorphic platforms. Thus, there are several reports on ion-gated EGTs for neuromorphic device applications.[18–20,91] For example, starch-based bio-polysaccharide electrolyte gated indium–gallium–zinc oxide (IGZO) photoelectric synergic coupled neuromorphic transistors have been reported, demonstrating potentials in visual perception–learning system applications.[92] Figure
In nervous system, synaptic plasticity is closely related to human brain learning and memory functions. In 1968, Atkinson and Shiffrin proposed a multistore model to explain the memory behavior.[94] The model describes three kinds of memories, including sensory memory (SM), short-term memory (STM), and long-term memory (LTM), as schematically shown in Fig.
Besides, there are two kinds of long-term synaptic plasticity includes long-term potentiation (LTP) and long-term depression (LTD).[96] The short-term synaptic plasticity provides temporary selection for neuron circuits, while LTP/LTD in synapses is the biological basis for continuous learning and memory. Thus, it is essential to mimic LTP and LTD to achieve cognitive complexity in artificial synapses. Qian et al.[97] achieved optical LTP and electrical LTD characteristics in optoelectronic synaptic device fabricated on a heterojunction consisted of copper-phthalocyanine (CuPc) and para-sexiphenyl (p-6P) layers. Figure
STDP plays a critical role in cognitive activities and determines the sign and magnitude of LTP or LTD. It refers to the change in synaptic weight with the interval time (Δtpre–post) between pre-synaptic and post-synaptic spikes and is considered to be a key feature of synaptic plasticity in biological nervous system.[98,99] STDP is a complement to Hebbian synaptic plasticity and is considered to be the basis for learning and memory by competitively strengthening and weakening of synapses in the neural network.[100] In canonical STDP, the strength of neuron connections will increase if pre-synaptic spike arrives before post-synaptic spike within a short interval time. Conversely, the strength of neuron connections will decrease if pre-synaptic spike arrives after post-synaptic spike within a short interval time. Thus, mimicking STDP is also highly important for neuromorphic system. By modifying the spike conditions, including spike duration and amplitude, spike shapes, different kinds of STDP behaviors have been mimicked on neuromorphic devices.[24,101–103] The coexistences of multiple types of synaptic plasticity enable the design of advanced neural structures. Yin et al.[55] reported several types of STDP behaviors on silicon nanocrystals (Si NCs) phototransistors through combinations of light and electrical stimuli. As shown in Fig.
In addition, STDP learning rules have been demonstrated with the synergistic effect of light and electrical stimuli on photoelectric synaptic devices based on 2D materials (MoS2,[18,82] black phosphorus,[53,54]), one-dimensional nanomaterials,[59] and metal–oxide–semiconductor materials.[61] Diverse learning rules can help artificial neural networks to deal with more complex situations and improve the learning efficiency.
In physiology, classic conditioning is a typical form of associative learning in nervous system.[104] Figure
Up to date, various synaptic characteristics have been achieved on a single artificial synaptic device. To build an integrated system of artificial neural system may be one important branch of future neuromorphic engineering. Visual perception system is an important sensory component in bionic electronics, which can adaptively detect, process, and memorize light signals.[108] Wang et al.[14] fabricated a ferroelectric/electrochemical modulated organic synapse and proposed an artificial visual-perception system. There are multiple organic functional layers in the device responsible for light-sensing. Thus, the device can transducer incident light signals into synaptic signals, both volatile and non-volatile. With the threshold property, color recognition function can be achieved. Synaptic signals triggered with light stimuli at wavelengths of 550 nm and 850 nm show different degrees of volatility. Thus, the devices demonstrate retinal functions. Lee et al.[109] proposed a visually sensor motor nervous system, as schematically shown in Fig.
Neuromorphic system could be a good way to overcome Von Neumann bottleneck. Recently, several kinds of neuromorphic devices, including resistance switching devices and transistors, have been proposed to mimic basic synaptic functions, such as short-term plasticity, long-term plasticity, synaptic filtering, spike-timing-dependent plasticity, metaplasticity, heteroplasticity, etc. Moreover, advanced neuron functions have been realized, including pattern memory and pattern recognition, classical conditions, and machine learning. All these demonstrate great potentials for solid-state neuromorphic devices in neuromorphic engineering.
In addition, developing visual perception-learning system at hardware level would also provide new opportunities for advanced intelligent learning system. In such a system, advanced intelligent photodetectors play the most important role for extension of human vision. Fortunately, progresses in advanced optoelectronic devices provide these possibilities. Neuromorphic devices possessing abilities to convert light stimuli into postsynaptic signals can also simplify the visual perception-learning system. Here, we review recent progress on optoelectronic neuromorphic memristors and optoelectronic neuromorphic transistors. These optoelectronic neuromorphic devices can mimic visual perception, information processing, and memory. Moreover, using synergistic stimuli in neuromorphic devices could significantly improve the diversity of signal inputs and the functional diversities of neural network systems and perception systems. It should be noted here that the integrated system of neuromorphic devices with artificial perception synapses and bionic electronic device with external sensory elements will have great potentials in wearable electronics, soft robots, and bionic prosthesis.
[1] | |
[2] | |
[3] | |
[4] | |
[5] | |
[6] | |
[7] | |
[8] | |
[9] | |
[10] | |
[11] | |
[12] | |
[13] | |
[14] | |
[15] | |
[16] | |
[17] | |
[18] | |
[19] | |
[20] | |
[21] | |
[22] | |
[23] | |
[24] | |
[25] | |
[26] | |
[27] | |
[28] | |
[29] | |
[30] | |
[31] | |
[32] | |
[33] | |
[34] | |
[35] | |
[36] | |
[37] | |
[38] | |
[39] | |
[40] | |
[41] | |
[42] | |
[43] | |
[44] | |
[45] | |
[46] | |
[47] | |
[48] | |
[49] | |
[50] | |
[51] | |
[52] | |
[53] | |
[54] | |
[55] | |
[56] | |
[57] | |
[58] | |
[59] | |
[60] | |
[61] | |
[62] | |
[63] | |
[64] | |
[65] | |
[66] | |
[67] | |
[68] | |
[69] | |
[70] | |
[71] | |
[72] | |
[73] | |
[74] | |
[75] | |
[76] | |
[77] | |
[78] | |
[79] | |
[80] | |
[81] | |
[82] | |
[83] | |
[84] | |
[85] | |
[86] | |
[87] | |
[88] | |
[89] | |
[90] | |
[91] | |
[92] | |
[93] | |
[94] | |
[95] | |
[96] | |
[97] | |
[98] | |
[99] | |
[100] | |
[101] | |
[102] | |
[103] | |
[104] | |
[105] | |
[106] | |
[107] | |
[108] | |
[109] | |
[110] |